Towards a calibrated problem set for Tik Tik

Cvetomir Dimov

Collaborative Problem Solving

1. What is it?

• a group goal that needs to be accomplished through problem solving
• a single individual cannot solve the problem alone
• no routine plan is available for the problem

Collaborative Problem Solving

2. How important is it?

One of the essential 21st century skills (according to AT21CS)

3. Why is it so important?

Many contemporary problems require teams of individuals with different expertise and background

Collaborative Planning

• envision alternative courses of action
• select and commit to one of these courses of action
• these can be jointly costructed and must be jointly agreed upon in the case of collaborative planning
• the problem difficulty is a function of, among others, how many alternative courses of action there are

Tik Tik: A Collaborative Planning Task

Project goal

1. Create a problem set, which is:

• within the general population’s abilities
• varies systematically in difficulty
• takes about an hour to complete

2. Identify level parameters that determine level difficulty

Designing problems

Factors that affect problem difficulty:

• number of barriers and number of barrier intersections
• start and goal locations

Varying problem difficulty systematically

• vary the minimum number of blockages necessary to solve a level (optimal solution length)
• vary how many unique courses of action there are in a level (search space size)

Problem creator and solver

Study

• 29 levels: 9 warmup levels + 20 main levels (between 1 and 5 blockages)
• 46 subjects (online platform Prolific)
• about 1.5 hours of total time
• there is a blockage limit and an optimal blockage number (i.e., optimal solution length) for each level
• subjects received 1 bonus point for a correct solution and 3 bonus points for an optimal solution

Subject performance

Problem difficulty

• Problems vary widely in difficulty
• More difficult problems take longer and are more rarely solved optimally

Which problem properties predict how difficult it is?

We still needs to find predictors of level difficulty, which include:

• optimal solution length - the minimum number of blockages needed to solve a level
• search space size - the number of possible sequences of blockages that can be taken
• trimmed search space size - the search space size up to length equal to that of the optimal solution

Predictors of level difficulty

• Optimal solution length: r = -0.67
• Search space size: r = -0.64
• Trimmed search space size: r = -0.82

A quick introduction to logistic regression

• The logistic model is an S-shaped curve that ranges between 0 and 1
• A logistic regression finds the logistic model that best describes a set of points

Logistic model prediction

Summary

• There are large individual differences in performance on Tik Tik
• Our problem set is within subject abilities, albeit a little difficult
• The trimmed search space size is the strongest predictor of level difficulty, together with the optimal solution length

Conclusion

• A callibrated problem set will enable us to measure collaborative planning performance precisely
• When combined with measures of various cognitive and social abilities, it will allow us to determine the relative contribution of these abilities.

Made with: Quil (problem solver), Noj libraries (data analysis), R (data analysis), Clay (data analysis + presentation)

source: notebooks/presentation.clj